Skip to content

feat: complete frontend UI batch filling, voice transcription & multilingual support#309

Open
utkarshqz wants to merge 3 commits intofireform-core:mainfrom
utkarshqz:feat/frontend-ui-voice-multilingual
Open

feat: complete frontend UI batch filling, voice transcription & multilingual support#309
utkarshqz wants to merge 3 commits intofireform-core:mainfrom
utkarshqz:feat/frontend-ui-voice-multilingual

Conversation

@utkarshqz
Copy link

Description

FireForm had a working backend — FastAPI endpoints, Ollama integration, PDF filling logic — but no frontend to use any of it. Officers had no way to interact with the system without writing raw API calls. This PR builds the complete interface that makes FireForm actually usable, and fixes the core extraction pipeline that was producing wrong results.


1. Complete Frontend UI

A full interface for FireForm's existing API — dark/light theme toggle, animated logo, drag-and-drop PDF upload, real-time API health indicator, session history, and responsive layout for station desk use.

SINGLE mode — select one template, describe the incident, download the filled PDF.

BATCH mode — check multiple agency templates, one transcript fills all simultaneously. Partial failures never abort the batch — each template result is independent.

Addresses: #1 #160 #173 #162
Fixes: #165 #205


2. O(M*N) → O(1) LLM Extraction

N : fields
M : Batch(multiple templates)

The original codebase called Ollama once per form field. Seven fields meant seven sequential blocking calls, ~35 seconds of wait, and hallucinations on two of seven fields. The model kept repeating the same answer because it received the same meaningless prompt seven times with no form context.

We replaced this with a single batch prompt. All fields sent simultaneously. Mistral sees the complete form structure and returns one JSON object.

Single form:
  Before: 7 fields → 7 calls → 152s → 2/7 hallucinated
  After:  7 fields → 1 call  →  59s → 7/7 correct

Batch (5 agency forms):
  Before: 5 forms × 7 fields → 35 calls → ~525s
  After:  5 forms × 7 fields →  1 call  →  ~60s
  LLM calls reduced: 35 → 1 regardless of batch size

Address: #69 raised by @shreyanshjain05
Fixes: #102 raised by @Acuspeedster


3. Batch Multi-Template Filling

POST /forms/fill/batch — one transcript fills every agency form simultaneously. Field discovery is dynamic: reads actual PDF annotation keys from the database. No hardcoded field lists. Works with any PDF from any agency, any country, without configuration.

For five templates: one LLM call total.

Fixes: #156 raised by @Acuspeedster
Fixes: #235


4. Lightweight PDF Field Extraction

PDF annotation reading via pypdf replaces vision-model parsing. Reads /TU (tooltip) and /T (technical name) from PDF annotations. Mistral receives "Full Name" instead of "textbox_0_0". Extraction accuracy improved without any model change.


5. Voice Transcription & Multilingual Support

Voice recording built directly into the transcript area. Officer records, stops, clicks Transcribe — transcript auto-fills into the textarea.

Multilingual: Auto-detects language. Works for non-English agencies — French, Spanish, Hindi, Arabic and 90+ languages supported natively. No configuration required.

Addresses: #44 raised by @pigeio


6. Schema Validation & Hallucination Detection

Field-level validation: phone format, email format, date format. Hallucination guard: same value appearing in 3+ different fields triggers rejection. Length guard: 500 character limit per field.

Fixes: #114


7. Checkbox & Radio Button Support

PDF checkboxes require PdfName values (/Yes, /Off) not strings. Old code wrote "yes" as a plain string — field stayed visually empty with no error. Fix reads the PDF's own appearance stream (AP.N) to find the exact on-state name per form. Works with any PDF. Properly handles grouped radio buttons with nested annotation structures, which often break when fields are treated as a flat list.

Fixes: #229


8. Progress Logging & Timeout Hardening

[LOG] per-field output with elapsed time. OLLAMA_TIMEOUT environment variable (default 120s). Explicit timeout error handling with clear error message.

Fixes: #68 #132


9. Test Suite — 70 Passing

Complete replacement of broken test suite. In-memory SQLite, proper fixtures, real HTTP client, no mocked path validation.

Fixes: #163 #182 #183 #186


Future Direction

  • PII masking — detect and tokenize sensitive fields before storage. Critical for law enforcement CJIS compliance. Audit trail text file alongside each PDF logging which transcript phrases produced which field values, with timestamps.
  • Multilingual extraction — prompt templates per language for non-English agencies
  • Offline-first field capture — PWA mobile companion for capturing data at scene without connectivity, syncing to station on return (separate PR)
  • Upgraded local models — larger quantized models (Llama 3, Phi-3) for better accuracy on complex forms, still CPU-only

Type of change

  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change
  • This change requires a documentation update

How Has This Been Tested?

  • Upload PDF, single fill, download — fields correctly filled
  • Batch mode with 2 templates simultaneously
  • Voice record → transcribe → form fill end to end
  • Multilingual voice (non-English transcript)
  • Checkbox and radio button filling on real fillable PDF
  • 503 when Ollama is down
  • 404 when template PDF missing from disk
  • Schema validation rejects hallucinated values
  • Dark/light theme toggle

Test Configuration:

  • Python: 3.11
  • OS: Windows 11
  • Ollama: local, mistral
  • Hardware: CPU-only, no GPU
python -m pytest tests/ -v
70 passed, 0 errors

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

This PR consolidates and builds on PRs #208 #209 #210 #213 #239 #241 #246.
@marcvergees @vharkins1 @juanalvv @jansans04 — happy to walk through any architectural decisions here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment